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Abstract Text - FPAR (2) : 

SOLUTION: The sight line positions are respectively 
determined by both of a 

pupil center method computation section 2D and cornea 
reflection method 

computation section 2E for the eyeball images of an 
operator M captured by an 

optical image pickup apparatus 1. When a detected result 
difference decision 

section 2F makes decision that the difference of the two 
detected LOS positions 

is not confined within a preset prescribed range 
continuously for the specified 

time, recomputation to determine the sight line positions 
again is carried out 

by the pupil center method computation section 2D according 
to the moving 

quantity of the head determined by a head moving quantity 
calculation section 

2G. As a result, the exact detection by the algorithm of 
the pupil center 

method is always carried out and, therefore, the exact 
sight line positions may 

be detected at all times without being affected by the 
rotating angle of the 

eyeballs and the mounting positions of illumination light 
for image pickup. 
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♦NOTICES* 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. * * * * shows the word which can not be translated. 
3 in the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] In the look detection approach of detecting the look location of an eyeball based on the 
eyeball image photoed by the optical image pick-up means The operation process in which a look 
detection result is obtained to the same eyeball image with each of both algorithms of the algorithm of a 
pupil core method, and the algorithm of a pupil-cornea reflection method, If the judgment result of the 
detection result difference judging process in which it judges whether it is restored to predetermined 
within the limits which the difference of the detection result by both algorithms set up beforehand, and a 
detection result difference judging process is no The look detection approach characterized by having 
the detection result re-operation process in which the re-operation for researching for the detection result 
by the algorithm of the pupil core method the error resulting from a motion of the head which has an 
eyeball for detection based on the detection result of the algorithm of a pupil-cornea reflection method 
was removed is performed. 

[Claim 2] In the look detection equipment constituted so that the look location of an eyeball might be 
detected based on the eyeball image photoed by the optical image pick-up means A pupil core method 
operation means to obtain a look detection result from an eyeball image according to the algorithm of a 
pupil core method, A cornea reflection method operation means to obtain a look detection result from an 
eyeball image according to the algorithm of a pupil-cornea reflection method, If the judgment result of a 
detection result difference judging means to judge whether it is restored to predetermined within the 
limits which the difference of the detection result by said both operation means set up beforehand, and a 
detection result difference judging means is no Look detection equipment characterized by having a 
detection result re-operation means to perform the re-operation for researching for the detection result by 
pupil core method operation means by which the error resulting from a motion of the head which has an 
eyeball for detection based on the detection result of a cornea reflection method operation means was 
removed 

[Claim 3] Look detection equipment constituted in look detection equipment according to claim 2 so 
that the detection result of a look location may be used as an object for the alter operation of a computer. 

[Claim 4] Look detection equipment constituted so that the look of an eyeball may be turned to said 
screen for alter operation of the head mount display which displays the screen for alter operation at least 
according to the command from a computer in look detection equipment according to claim 3. 



[Translation done.] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the approach and equipment which detect the look 
location (the direction of a look) of an eyeball based on the eyeball image captured using the optical 
image pick-up means, especially relates to a technique to always detect the look location where the 
eyeball for detection is exact. 
[0002] 

[Description of the Prior Art] As an approach of detecting the look location of an eyeball, it is usually 
supposed that a pupil core method and a pupil-cornea reflection method are especially useful. A pupil 
core method is the technique of using that a pupil core moves according to rotation of the eyeball by 
change of a look location, measuring the center position of a pupil, and ****(ing) a look location 
simply, the pupil-cornea reflection method uses that the virtual image (reflected image) produced in the 
light reflex on the front face of a cornea moves according to rotation of the eyeball by change of a look 
location, measures the center position and pupil center position of a virtual image on the front face of a 
cornea, and **** a look location from the relation of both locations - it is technique. 
[0003] It faces actually ****(ing) a look location, and the eyeball for detection is photoed with an 
optical image pick-up means, the eyeball image is captured, and a look location is determined by 
analyzing the eyeball image captured according to the algorithm of a pupil core method or a pupil- 
cornea reflection method. 

[0004] Since the movement magnitude of the pupil center position accompanying change of a look 
location is large in the case of a pupil core method, there is an advantage of being easy to detect a look 
location. Since migration of the pupil center position by rotation of an eyeball and migration of the pupil 
center position by motion of a head are separable to some extent in the case of a pupil-cornea reflection 
method, even if the head is not being fixed completely, there i s an advantage that a look location is 
detectable. 
[0005] 

[Problem(s) to be Solved by the Invention] However, there is a problem that the look location where the 

eyeball for detection is exact cannot always detect by the detection approach by the above-mentioned 

pupil core method or the detection approach by the pupil-cornea reflection method, either. 

[0006] That is, since migration of the pupil center position by rotation of an eyeball and migration of the 

pupil center position by motion of a head are inseparable even if it is easy to carry out detection in the 

case of the detection approach by the pupil core method, when there is a motion of a head, there is a 

problem that this cannot serve as an error and cannot detect a look location correctly. 

[0007] Moreover, in the case of the detection approach by the pupil-cornea reflection method, a virtual 

image is not formed in a cornea front face depending on the installation location of the angle of rotation 

of an eyeball, or the illumination light for an image pick-up, but there is a problem that a look location is 

undetectable. 

[0008] This invention makes it a technical problem to offer the look detection approach which the look 
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location where the eyeball for detection is exact can always detect, and its equipment in view of the 

above-mentioned situation. 

[0009] 

[Means for Solving the Problem] In order to solve said technical problem, the look detection approach 
concerning invention of claim 1 In the look detection approach of detecting the look location of an 
eyeball based on the eyeball image photoed by the optical image pick-up means The operation process 
in which a look detection result is obtained to the same eyeball image with each of both algorithms of 
the algorithm of a pupil core method, and the algorithm of a pupil-cornea reflection method, If the 
judgment result of the detection result difference judging process in which it judges whether it is 
restored to predetermined within the limits which the difference of the detection result by both 
algorithms set up beforehand, and a detection result difference judging process is no It is characterized 
by having the detection result re-operation process in which the re-operation for researching for the 
detection result by the algorithm of the pupil core method the error resulting from a motion of the head 
which has an eyeball for detection based on the detection result of the algorithm of a pupil-cornea 
reflection method was removed is performed. 

[0010] Furthermore, the look detection equipment concerning invention of claim 2 In the look detection 
equipment constituted so that the look location of an eyeball might be detected based on the eyeball 
image photoed by the optical image pick-up means A pupil core method operation means to obtain a 
look detection result from an eyeball image according to the algorithm of a pupil core method, A cornea 
reflection method operation means to obtain a look detection result from an eyeball image according to 
the algorithm of a pupil-cornea reflection method, If the judgment result of a detection result difference 
judging means to judge whether it is restored to predetermined within the limits which the difference of 
the detection result by said both operation means set up beforehand, and a detection result difference 
judging means is no It is characterized by having a detection result re-operation means to perform the re- 
operation for researching for the detection result by pupil core method operation means by which the 
error resulting from a motion of the head which has an eyeball for detection based on the detection result 
of a cornea reflection method operation means was removed. 

[001 1] Moreover, invention of claim 3 is characterized by constituting the detection result of a look 
location so that it may be used as an object for the alter operation of a computer in look detection 
equipment according to claim 2. 

[0012] Moreover, invention of claim 4 is characterized by being constituted so that the look of an 
eyeball may be turned to said screen for alter operation of the head mount display which displays the 
screen for alter operation at least according to the command from a computer in look detection 
equipment according to claim 3. 

[0013] The operation in [Function], next the look detection equipment concerning this invention is 
explained. When the look location (the direction of a look) of an eyeball is detected by the approach of 
this invention, After photoing first the eyeball image of the eyeball which is a candidate for detection 
with an optical image pick-up means and incorporating Next, after obtaining two look detection results 
to the captured eyeball image by the operation by each of both algorithms of the algorithm of a pupil 
core method, and the algorithm of a pupil-cornea reflection method, the difference of two obtained 
detection results judges whether it is restored to predetermined within the limits set up beforehand. 
When the difference of the detection result of both algorithms is not restored to predetermined within the 
limits (in the case of negation), the error will have arisen in the detection result by the algorithm of a 
pupil core method by migration of the pupil center position by motion of a head with the eyeball for 
detection. 

[0014] Then, in the approach of this invention, if a judgment result is negation, the detection result by 
the algorithm of the pupil core method the error resulting from a motion of the head which a re- 
operation is performed based on the detection result of the algorithm of a pupil-cornea reflection 
method, and has an eyeball for detection was removed will be researched for, and an exact look location 
will be ****(ed). That is, since migration of the pupil center position by rotation of an eyeball and 
migration of the pupil center position by motion of the head which has the eyeball for detection are 
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separable in the case of the algorithm of a pupil-cornea reflection method, the movement magnitude of 
the pupil center position by motion of a head is calculated, and the error resulting from a motion of a 
head is removed by amending the pupil center position in the algorithm of a pupil core method using 
this. 

[0015] In addition, when the difference of the detection result of both algorithms is restored to 
predetermined within the limits (in the case of affirmation), each detection result of the look location by 
both the algorithms of a pupil core method and a pupil-cornea reflection method will be called accuracy. 

[0016] Therefore, in the look detection approach of this invention, the look location where an eyeball is 
exact can always detect, without being influenced by the installation location of the angle of rotation of 
an eyeball, or the illumination light for an image pick-up since the exact detection result by the 
algorithm of a pupil core method is always obtained. 

[0017] Furthermore, in the case of the equipment of invention of claim 2, the look location of the eyeball 
for detection is always correctly detectable by inventing claim 1. 

[0018] Moreover, in the case of the look detection equipment of claim 3, according to the look location 

always detected correctly, alter operation of a computer is performed always rightly. 

[0019] Moreover, in the case of the look detection equipment of claim 4, the input for computer 

operation is performed from the screen for alter operation on display as it is also with the look of the 

eyeball turned to the screen for alter operation displayed on a head mount display according to the 

command from a computer (computer operator). 

[0020] 

[Embodiment of the Invention] Then, one example of this invention is explained, referring to a drawing. 
Drawing 1 is the block diagram showing the whole computer communication system configuration 
which used an example of the look detection equipment of claims 2-4. Moreover, the look detection 
equipment currently arranged into the system of drawing 1 is equipment which detects a look location 
(the direction of a look) by carrying out an example of the look detection approach of claim 1. 
[0021] The computer communication system of an example So that the look location of an eyeball may 
be detected based on Operator's M eyeball image photoed by the optical image pick-up device 1 which 
photos Operator's M eyeball image, and the optical image pick-up device 1 Configuration **** look 
detection equipment 2, The personal computer 3 with which alter operation is made according to the 
detection result of the look location by look detection equipment 2, It has the head mount display 
(HMD) 4 with which the screen for alter operation is displayed at least according to the command from 
a personal computer 3. While the look location of Operator's M eyeball turned to the screen for alter 
operation of a head mount display 4 is detected by look detection equipment 2 It has the composition 
that alter operation is made to a personal computer 3 according to the detected look location, and 
communication is performed between Operator M and a computer 3. 

[0022] Hereafter, the configuration of each part of the computer communication system of an example is 
explained concretely. 

[0023] The optical image pick-up device 1 consists of source of infrared light 1A, half mirror IB, and 
CCD camera 1C which are installed in the head mount display 4, and while Operator's M eyeball 
illuminated with the exposure light of source of infrared light 1 A is photoed by CCD camera 1C through 
half mirror IB, the photoed eyeball image has composition sent out to look detection equipment 2 from 
CCD camera 1C. In addition, the optical image pick-up device 1 is formed one pair in right-and-left both 
eyes. 

[0024] LCD(liquid crystal display)4A which provides the operator M putting on a head mount display 4 
with the screen for alter operation, and optical-system 4for display B and half mirror 4C of a pair were 
prepared in right-and-left both eyes, respectively, a picture signal is offered [ well-known ] from a 
personal computer 3 at LCD4A, and the enlarged display of the screen for alter operation is carried out 
to the view of the operator M who equips. Images, such as an actuation key and an icon, project on the 
screen for alter operation at a position. Since in the case of the enlarged display by the head mount 
display 4 an angle of visibility is expanded and the movement magnitude of the look of the operator at 
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the time of migration of a fixation point becomes large, there is an advantage whose detection precision 
of a look location improves. 

[0025] 3 value-ized circuit 2A which look detection equipment 2 3 -value— ization-processes the captured 
eyeball image according to the "threshold" set up beforehand, and obtains 3 value-ized image, If it has 
pupil extract section 2B which **** a pupil center position, and reflected image extract section 2C 
which **** the location (core) of the reflected image (virtual image) produced from 3 value-ized image 
on a cornea front face from 3 value-ized image and one eyeball image is newly captured It has the 
composition that a pupil center position and the location of a reflected image are ****(ed) immediately. 
In addition, in 3 value-ized image, the darkest field is a pupil image and the brightest field corresponds 
to a reflected image. 

[0026] Furthermore, look detection equipment 2 is equipped with pupil core method operation part 2D 
which **** a look location according to the algorithm of a pupil core method from an eyeball image, 
and cornea reflection method operation part 2E which **** a look location according to the algorithm of 
a pupil-cornea reflection method from an eyeball image. That is, pupil core method operation part 2D 
computes a look location comparatively simply from the pupil center position called for by pupil extract 
section 2B, and cornea reflection method operation part 2E computes a look location from the pupil 
center position called for by pupil extract section 2B, and the location of the reflected image searched 
for by reflected image extract section 2C. 

[0027] The look location computed by both [ these ] operation part 2D and 2E corresponds to the screen 
for alter operation displayed in a head mount display 4. That is, the calibration of a look location is 
beforehand performed to the standard point of the screen for alter operation, and the look location 
computed by both operation part 2D and 2E has composition matched with the fixation point on the 
screen for alter operation. 

[0028] As shown in drawing 2 , after specifically asking for the pupil center position on an eyeball 
image in case an eyeball gazes at standard point Pa-Pd which is four of the four directions whose 
locations of Screen PA for alter operation are known in order and a look is in each of each standard 
point Pa-Pd, Furthermore the correlation of the coordinate and pupil center position of four standard 
point Pa-Pd in Screen PA for alter operation, a pupil center position, and a reflected image location is 
searched for beforehand, and the pupil center position at the time of an observation is changed into the 
look location which corresponds on the screen for alter operation according to this correlation. In 
addition, in the case of the screen for alter operation of an example, it has composition which shows a 
look location by XY rectangular coordinates which make the upper left a zero. 
[0029] And it has detection result difference judging section 2F which judge whether look detection 
equipment 2 carried out fixed time amount continuation at predetermined within the limits which the 
difference of the detection result of both algorithms set up beforehand, and it is settled. In negation of 
the judgment result by detection result difference judging section 2F (not restored to predetermined 
within the limits), in the look location computed by pupil core method operation part 2D, the error 
resulting from a motion of Operator's M head is included, and a detection result will not be exact. 
[0030] On the contrary, in affirmation of the judgment result by detection result difference judging 
section 2F (restored to predetermined within the limits), there is no motion in Operator's M head, and the 
error resulting from a motion of Operator's M head will not be included in the look location computed 
by both operation part 2D and 2E, but the detection result will be exact. In this case, the look location 
computed by pupil core method operation part 2D is sent out immediately to a personal computer 3. 
[0031] Furthermore, look detection equipment 2 is equipped with head movement magnitude calculation 
section 2G which compute the movement magnitude of Operator's M head from the detection result by 
cornea reflection method operation part 2E. That is, since migration of the pupil center position by 
rotation of an eyeball and migration of the pupil center position by motion of a head are separable in 
general in the case of a pupil-cornea reflection method When the judgment result by detection result 
difference judging section 2F is negation, head movement magnitude calculation section 2G It has 
composition which computes the movement magnitude (on an eyeball image) (del tax, delta y) of the 
pupil center position by motion of a head in the form [ like ] which carries out inverse transformation 
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processing of the detection result by cornea reflection method operation part 2E, and is sent into pupil 
core method operation part 2D. 

[0032] As shown in drawing 3 , when it is more specifically pupil center position Ps= (xc, yc) on the 
eyeball image PB in the case of pupil core method operation part 2D, as shown in drawing 2 , according 
to the following operation expression (1), look location Pt= (Xw, Yw) in Screen PA for alter operation is 
computed. The matrix which uses ABCD of operation expression (1) as an element corresponds to the 
correlation between the pupil center position Ps on the eyeball image PB called for as mentioned above 
based on four standard point Pa-Pd in Screen PA for alter operation, and the look location Pt on Screen 
PA for alter operation (correspondence relation). 
0033] 
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[0034] When movement magnitude deltaP= (deltax, delta y) of the pupil center position by motion of a 
head is sent into pupil core method operation part 2D from head movement magnitude calculation 
section 2G, however, the pupil center position Ps Since only the movement magnitude deltaP of a pupil 
center position will have shifted from original pupil center position Pu= (xr, yr) according to the amount 
by which the head moved, head movement magnitude calculation section 2G The re-operation which 
computes the exact look location PT (Xv, Yv) where gap of movement magnitude deltaP was taken into 
consideration, and the error by motion of a head was eliminated according to the following operation 
expression (2) will be performed. In addition, it is xr=xc+delta x and yr=yc+deltay and movement 
magnitude deltaP= (deltax, delta y) becomes forward or a negative-number value according to the sense 
of a coordinate. 

[0035] In addition, the look location for which it reasked by pupil core method operation part 2D is sent 
out to a personal computer 3. Therefore, in the case of the example, a detection result re-operation 
means to research for the detection result by pupil core method operation means by which the error 
resulting from a motion of the head which has an eyeball for detection based on the detection result of 
cornea reflection method operation part 2E was removed will consist of head movement magnitude 
calculation section 2G and pupil core method operation part 2D. 
0036] 
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[0037] Moreover, look detection equipment 2 is equipped with blink detection section 2H which detect 
Operator M having winked and having carried out from the detection result by pupil core method 
operation part 2D, and blink detection section 2H send out the signal which winks to a personal 
computer 3 immediately and tells generating, if the blink of Operator's M eye is detected. 
[0038] On the other hand, according to the look location sent out from pupil core method operation part 
2D, a personal computer 3 indicates by superposition the cursor-like marker mk who shows a look 
location to Screen PA for alter operation of the head UMANTO display 4, as shown in drawing 4 . 
Marker mk has composition which moves in the Screen PA top for alter operation according to look 
change of Operator M. On the other hand, if the signal which a personal computer 3 winks from blink 
detection section 2H, and tells generating is received, as shown in drawing 5 , it will be judged that alter 
operation was performed to the actuation key (or icon) nk in which Marker mk is located. 
[0039] thus, since in the case of the system of an example Operator M winks with a look, it comes out 
and alter operation of a personal computer 3 can be performed, it becomes possible to communicate with 
a personal computer 3, while the serious trouble back tone who cannot do actuation of a keyboard at all 
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had also gone to sleep (dialogue). 

[0040] Then, the activation process of communication by the system of the example which has the 
configuration described above is explained, referring to a drawing. Drawing 6 is a flow chart which 
shows the advance situation of communication by the system of an example. 
[0041] In addition, below, after Operator M is equipped with a head mount display 4, it explains as a 
thing in the phase which the matrix which uses ABCD as a multiplier equivalent to the correlation of the 
look location and pupil center position on Screen PA for alter operation as an element was already **** 
(ed), and all preparations finished. 

[0042] [Step SI] Operators M eyeball is photoed by the optical image pick-up device 1, and an eyeball 
image is sent into look detection equipment 2. 

[0043] [Step S2] After the eyeball image captured by 3 value-ized circuit 2A is formed into 3 values, 
while a pupil center position is called for by pupil extract section 2B from 3 value-ized image, the 
location of a reflected image is called for by reflected image extract section 2B. 
[0044] [Step S3] While a look location is called for from a pupil center position by pupil core method 
operation part 2D, a look location is detected by corneal-reflex operation part 2E from a pupil center 
position and the location of a reflected image. 

[0045] [Step S4] The difference of both operation part 2D and the detection result of the look location of 
2E carries out fixed time amount continuation by detection result difference judging section 2F, and it is 
judged whether it is predetermined within the limits. If a judgment result is affirmation, it will fly to step 
S7, and if a judgment result is negation, it will progress to the following step S5. 
[0046] [Step S5] Movement magnitude deltaP based on [ by motion of Operator's M head ] pupils is 
****(ed) by head movement magnitude calculation section 2G. 

[0047] [Step S6] Pupil core method operation part 2D takes into consideration movement magnitude 
deltaP based on [ by motion of Operator's M head ] pupils, and re**** the look location PT. 
[0048] [Step S7] The look location Pt or the look location PT ****(ed) by pupil core method operation 
part 2D is sent into a personal computer 3. 

[0049] [Step S8] When a blink of Operator M is detected by blink detecting-element 2H, it progresses to 
the following step S9, and when a blink of Operator M is not detected, it flies to step S10. 
[0050] [Step S9] The signal which winks from blink detecting-element 2H, and tells generating is sent 
into a personal computer 3, and alter operation corresponding to a look location is performed. 
[0051] [Step S10] When continuing detection of a look location, return and the following steps are 
repeated to step SI. Communication is ended when not continuing detection of a look location. 
[0052] It is not restored to predetermined within the limits which the difference of each two detection 
results computed by each algorithm of a pupil core method and a pupil-cornea reflection method to the 
captured eyeball image in the example set up beforehand as stated above. When the error resulting from 
a motion of Operator's M head exists, it has the composition that the re-operation which reasks for the 
look location except the error which originates in a motion of Operator's M head in quest of movement 
magnitude deltaP of the pupil center position by motion of a head is performed. Consequently, exact 
detection by the algorithm of a pupil core method is always performed, and the look location where 
Operator's M eyeball is exact can always detect, without being influenced by the installation location of 
the angle of rotation of an eyeball, or the illumination light for an image pick-up. 
[0053] This invention is not restricted to the gestalt of the above-mentioned implementation, and can 
carry out deformation implementation as follows. 

[0054] (1) If required location detection is possible, you may make it adopt in the example, the 
configuration which binary-ization-processes an eyeball image, although it was the configuration of 3- 
value—izati on-processing an eyeball image. 

[0055] (2) In the example, although it was the configuration of deciding a look location because there 
was a blink, the thing of a configuration of deciding a look location by switch actuation or look duration 
is mentioned as a modification. 

[0056] (3) In the case of the example, it was the configuration that the detection result of a look location 
was used for the alter operation of a computer, but in this invention, the detection result of a look 
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location may be a configuration used for things other than the alter operation of a computer, and may be 

only a configuration which only searches for the detection result of a look location. 

[0057] 

[Effect of the Invention] As explained in full detail above, while obtaining a 2 look detection result to 
the eyeball image captured with the optical image pick-up means with each of both algorithms of the 
algorithm of a pupil core method, and the algorithm of a pupil-cornea reflection method according to the 
look detection approach of invention of claim 1 When the error resulting from a motion of the head 
where the difference of two detection results is not restored to predetermined within the limits set up 
beforehand, and the eyeball for detection has it arises It has the configuration from which the detection 
result by the algorithm of the pupil core method the error resulting from a motion of the head which a re- 
operation is performed based on the detection result of the algorithm of a pupil-cornea reflection 
method, and has an eyeball for detection was removed is obtained. The look location where the eyeball 
for detection is exact can always detect, without being influenced by the installation location of the 
angle of rotation of an eyeball, or the illumination light for an image pick-up since exact detection by the 
algorithm of a pupil core method is always performed. 

[0058] Furthermore, according to the look detection equipment of invention of claim 2, the look location 
where the eyeball for detection is exact is detectable by enforcing the approach of invention of claim L 
[0059] Moreover, since the look location always detected correctly has composition used as an object 
for the alter operation of a computer according to the look detection equipment of invention of claim 3, 
alter operation of a computer is performed always rightly. 
[0060] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

l.This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2 **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



TECHNICAL FIELD 



[Field of the Invention] This invention relates to the approach and equipment which detect the look 
location (the direction of a look) of an eyeball based on the eyeball image captured using the optical 
image pick-up means, especially relates to a technique to always detect the look location where the 
eyeball for detection is exact. 



[Translation done.] 
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PRIOR ART 



Pescription of the Prior Art] As an approach of detecting the look location of an eyeball, it is usually 
supposed that a pupil core method and a pupil-cornea reflection method are especially useful. A pupil 
core method is the technique of using that a pupil core moves according to rotation of the eyeball by 
change of a look location, measuring the center position of a pupil, and ****(ing) a look location 
simply, the pupil-cornea reflection method uses that the virtual image (reflected image) produced in the 
light reflex on the front face of a cornea moves according to rotation of the eyeball by change of a look 
location, measures the center position and pupil center position of a virtual image on the front face of a 
cornea, and **** a look location from the relation of both locations — it is technique. 
[0003] It faces actually ****(ing) a look location, and the eyeball for detection is photoed with an 
optical image pick-up means, the eyeball image is captured, and a look location is determined by 
analyzing the eyeball image captured according to the algorithm of a pupil core method or a pupil- 
cornea reflection method. 

[0004] Since the movement magnitude of the pupil center position accompanying change of a look 
location is large in the case of a pupil core method, there is an advantage of being easy to detect a look 
location. Since migration of the pupil center position by rotation of an eyeball and migration of the pupil 
center position by motion of a head are separable to some extent in the case of a pupil-cornea reflection 
method, even if the head is not being fixed completely, there is an advantage that a look location is 
detectable. 
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EFFECT OF THE INVENTION 



[Effect of the Invention] As explained in full detail above, while obtaining a 2 look detection result to 
the eyeball image captured with the optical image pick-up means with each of both algorithms of the 
algorithm of a pupil core method, and the algorithm of a pupil-cornea reflection method according to the 
look detection approach of invention of claim 1 When the error resulting from a motion of the head 
where the difference of two detection results is not restored to predetermined within the limits set up 
beforehand, and the eyeball for detection has it arises It has the configuration from which the detection 
result by the algorithm of the pupil core method the error resulting from a motion of the head which a re- 
operation is performed based on the detection result of the algorithm of a pupil-cornea reflection 
method, and has an eyeball for detection was removed is obtained. The look location where the eyeball 
for detection is exact can always detect, without being influenced by the installation location of the 
angle of rotation of an eyeball, or the illumination light for an image pick-up since exact detection by the 
algorithm of a pupil core method is always performed. 

[0058] Furthermore, according to the look detection equipment of invention of claim 2, the look location 
where the eyeball for detection is exact is detectable by enforcing the approach of invention of claim 1. 
[0059] Moreover, since the look location always detected correctly has composition used as an object 
for the alter operation of a computer according to the look detection equipment of invention of claim 3, 
alter operation of a computer is performed always rightly. 

[0060] Moreover, since it has the composition that the look of an eyeball is turned to the screen for alter 
operation displayed on a head mount display according to the command from a computer according to 
the look detection equipment of invention of claim 4, it becomes possible from the screen for alter 
operation on display to a head mount display to perform the input for computer operation as it is also 
with the look of an eyeball. 
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TECHNICAL PROBLEM 



[Problem(s) to be Solved by the Invention] However, there is a problem that the look location where the 

eyeball for detection is exact cannot always detect by the detection approach by the above-mentioned 

pupil core method or the detection approach by the pupil -cornea reflection method, either. 

[0006] That is, since migration of the pupil center position by rotation of an eyeball and migration of the 

pupil center position by motion of a head are inseparable even if it is easy to carry out detection in the 

case of the detection approach by the pupil core method, when there is a motion of a head, there is a 

problem that this cannot serve as an error and cannot detect a look location correctly. 

[0007] Moreover, in the case of the detection approach by the pupil-cornea reflection method, a virtual 

image is not formed in a cornea front face depending on the installation location of the angle of rotation 

of an eyeball, or the illumination light for an image pick-up, but there is a problem that a look location is 

undetectable. 

[0008] This invention makes it a technical problem to offer the look detection approach which the look 
location where the eyeball for detection is exact can always detect, and its equipment in view of the 
above-mentioned situation. 
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MEANS 



[Means for Solving the Problem] In order to solve said technicial problem, the look detection approach 
concerning invention of claim 1 In the look detection approach of detecting the look location of an 
eyeball based on the eyeball image photoed by the optical image pick-up means The operation process 
in which a look detection result is obtained to the same eyeball image with each of both algorithms of 
the algorithm of a pupil core method, and the algorithm of a pupil-cornea reflection method, If the 
judgment result of the detection result difference judging process in which it judges whether it is 
restored to predetermined within the limits which the difference of the detection result by both 
algorithms set up beforehand, and a detection result difference judging process is no It is characterized 
by having the detection result re-operation process in which the re-operation for researching for the 
detection result by the algorithm of the pupil core method the error resulting from a motion of the head 
which has an eyeball for detection based on the detection result of the algorithm of a pupil-cornea 
reflection method was removed is performed. 

[0010] Furthermore, the look detection equipment concerning invention of claim 2 In the look detection 
equipment constituted so that the look location of an eyeball might be detected based on the eyeball 
image photoed by the optical image pick-up means A pupil core method operation means to obtain a 
look detection result from an eyeball image according to the algorithm of a pupil core method, A cornea 
reflection method operation means to obtain a look detection result from an eyeball image according to 
the algorithm of a pupil-comea reflection method, If the judgment result of a detection result difference 
judging means to judge whether it is restored to predetermined within the limits which the difference of 
the detection result by said both operation means set up beforehand, and a detection result difference 
judging means is no It is characterized by having a detection result re-operation means to perform the re- 
operation for researching for the detection result by pupil core method operation means by which the 
error resulting from a motion of the head which has an eyeball for detection based on the detection result 
of a cornea reflection method operation means was removed. 

[001 1] Moreover, invention of claim 3 is characterized by constituting the detection result of a look 
location so that it may be used as an object for the alter operation of a computer in look detection 
equipment according to claim 2. 

[0012] Moreover, invention of claim 4 is characterized by being constituted so that the look of an 
eyeball may be turned to said screen for alter operation of the head mount display which displays the 
screen for alter operation at least according to the command from a computer in look detection 
equipment according to claim 3. 
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OPERATION 



The operation in [Function], next the look detection equipment concerning this invention is explained. 
When the look location (the direction of a look) of an eyeball is detected by the approach of this 
invention, After photoing first the eyeball image of the eyeball which is a candidate for detection with an 
optical image pick-up means and incorporating Next, after obtaining two look detection results to the 
captured eyeball image by the operation by each of both algorithms of the algorithm of a pupil core 
method, and the algorithm of a pupil-cornea reflection method, the difference of two obtained detection 
results judges whether it is restored to predetermined within the limits set up beforehand. When the 
difference of the detection result of both algorithms is not restored to predetermined within the limits (in 
the case of negation), the error will have arisen in the detection result by the algorithm of a pupil core 
method by migration of the pupil center position by motion of a head with the eyeball for detection. 
[0014] Then, in the approach of this invention, if a judgment result is negation, the detection result by 
the algorithm of the pupil core method the error resulting from a motion of the head which a re- 
operation is performed based on the detection result of the algorithm of a pupil-cornea reflection 
method, and has an eyeball for detection was removed will be researched for, and an exact look location 
will be ****(ed). That is, since migration of the pupil center position by rotation of an eyeball and 
migration of the pupil center position by motion of the head which has the eyeball for detection are 
separable in the case of the algorithm of a pupil-cornea reflection method, the movement magnitude of 
the pupil center position by motion of a head is calculated, and the error resulting from a motion of a 
head is removed by amending the pupil center position in the algorithm of a pupil core method using 
this. 

[0015] In addition, when the difference of the detection result of both algorithms is restored to 
predetermined within the limits (in the case of affirmation), each detection result of the look location by 
both the algorithms of a pupil core method and a pupil-cornea reflection method will be called accuracy. 

[0016] Therefore, in the look detection approach of this invention, the look location where an eyeball is • 
exact can always detect, without being influenced by the installation location of the angle of rotation of 
an eyeball, or the illumination light for an image pick-up since the exact detection result by the 
algorithm of a pupil core method is always obtained. 

[0017] Furthermore, in the case of the equipment of invention of claim 2, the look location of the eyeball 
for detection is always correctly detectable by inventing claim 1. 

[0018] Moreover, in the case of the look detection equipment of claim 3, according to the look location 

always detected correctly, alter operation of a computer is performed always rightly. 

[0019] Moreover, in the case of the look detection equipment of claim 4, the input for computer 

operation is performed from the screen for alter operation on display as it is also with the look of the 

eyeball turned to the screen for alter operation displayed on a head mount display according to the 

command from a computer (computer operator). 

[0020] 

[Embodiment of the Invention] Then, one example of this invention is explained, referring to a drawing. 
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Drawing 1 is the block diagram showing the whole computer communication system configuration 
which used an example of the look detection equipment of claims 2-4. Moreover, the look detection 
equipment currently arranged into the system of drawing 1 is equipment which detects a look location 
(the direction of a look) by carrying out an example of the look detection approach of claim 1. 
[0021] The computer communication system of an example So that the look location of an eyeball may 
be detected based on Operators M eyeball image photoed by the optical image pick-up device 1 which 
photos Operator's M eyeball image, and the optical image pick-up device 1 Configuration **** look 
detection equipment 2, The personal computer 3 with which alter operation is made according to the • 
detection result of the look location by look detection equipment 2, It has the head mount display 
(HMD) 4 with which the screen for alter operation is displayed at least according to the command from 
a personal computer 3. While the look location of Operator's M eyeball turned to the screen for alter 
operation of a head mount display 4 is detected by look detection equipment 2 It has the composition 
that alter operation is made to a personal computer 3 according to the detected look location, and 
communication is performed between Operator M and a computer 3. 

[0022] Hereafter, the configuration of each part of the computer communication system of an example is 
explained concretely. 

[0023] The optical image pick-up device 1 consists of source of infrared light 1A, half mirror IB, and 
CCD camera 1C which are installed in the head mount display 4, and while Operator's M eyeball 
illuminated with the exposure light of source of infrared light 1A is photoed by CCD camera 1C through 
half mirror IB, the photoed eyeball image has composition sent out to look detection equipment 2 from 
CCD camera 1C. In addition, the optical image pick-up device 1 is formed one pair in right-and-left both 
eyes. 

[0024] LCD(liquid crystal display )4A which provides the operator M putting on a head mount display 4 
with the screen for alter operation, and optical-system 4for display B and half mirror 4C of a pair were 
prepared in right-and-left both eyes, respectively, a picture signal is offered [ well-known ] from a 
personal computer 3 at LCD4A, and the enlarged display of the screen for alter operation is carried out 
to the view of the operator M who equips. Images, such as an actuation key and an icon, project on the 
screen for alter operation at a position. Since in the case of the enlarged display by the head mount 
display 4 an angle of visibility is expanded and the movement magnitude of the look of the operator at 
the time of migration of a fixation point becomes large, there is an advantage whose detection precision 
of a look location improves. 

[0025] 3 value-ized circuit 2 A which look detection equipment 2 3-value-ization-processes the captured 
eyeball image according to the "threshold" set up beforehand, and obtains 3 value-ized image, If it has 
pupil extract section 2B which **** a pupil center position, and reflected image extract section 2C 
which **** the location (core) of the reflected image (virtual image) produced from 3 value-ized image 
on a cornea front face from 3 value-ized image and one eyeball image is newly captured It has the 
composition that a pupil center position and the location of a reflected image are ****(ed) immediately. 
In addition, in 3 value-ized image, the darkest field is a pupil image and the brightest field corresponds 
to a reflected image. 

[0026] Furthermore, look detection equipment 2 is equipped with pupil core method operation part 2D 
which **** a look location according to the algorithm of a pupil core method from an eyeball image, 
and cornea reflection method operation part 2E which **** a look location according to the algorithm of 
a pupil-cornea reflection method from an eyeball image. That is, pupil core method operation part 2D 
computes a look location comparatively simply from the pupil center position called for by pupil extract 
section 2B, and cornea reflection method operation part 2E computes a look location from the pupil 
center position called for by pupil extract section 2B, and the location of the reflected image searched 
for by reflected image extract section 2C. 

[0027] The look location computed by both [ these ] operation part 2D and 2E corresponds to the screen 
for alter operation displayed in a head mount display 4. That is, the calibration of a look location is 
beforehand performed to the standard point of the screen for alter operation, and the look location 
computed by both operation part 2D and 2E has composition matched with the fixation point on the 
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screen for alter operation. 

[0028] As shown in drawing 2 , after specifically asking for the pupil center position on an eyeball 
image in case an eyeball gazes at standard point Pa-Pd which is four of the four directions whose 
locations of Screen PA for alter operation are known in order and a look is in each of each standard 
point Pa-Pd, Furthermore the correlation of the coordinate and pupil center position of four standard 
point Pa-Pd in Screen PA for alter operation, a pupil center position, and a reflected image location is 
searched for beforehand, and the pupil center position at the time of an observation is changed into the 
look location which corresponds on the screen for alter operation according to this correlation. In 
addition, in the case of the screen for alter operation of an example, it has composition which shows a 
look location by XY rectangular coordinates which make the upper left a zero. 
[0029] And it has detection result difference judging section 2F which judge whether look detection 
equipment 2 carried out fixed time amount continuation at predetermined within the limits which the 
difference of the detection result of both algorithms set up beforehand, and it is settled. In negation of 
the judgment result by detection result difference judging section 2F (not restored to predetermined 
within the limits), in the look location computed by pupil core method operation part 2D, the error 
resulting from a motion of Operator's M head is included, and a detection result will not be exact. 
[0030] On the contrary, in affirmation of the judgment result by detection result difference judging 
section 2F (restored to predetermined within the limits), there is no motion in Operator's M head, and the 
error resulting from a motion of Operator's M head will not be included in the look location computed 
by both operation part 2D and 2E, but the detection result will be exact. In this case, the look location 
computed by pupil core method operation part 2D is sent out immediately to a personal computer 3. 
[0031] Furthermore, look detection equipment 2 is equipped with head movement magnitude calculation 
section 2G which compute the movement magnitude of Operator's M head from the detection result by 
cornea reflection method operation part 2E. That is, since migration of the pupil center position by 
rotation of an eyeball and migration of the pupil center position by motion of a head are separable in 
general in the case of a pupil-cornea reflection method When the judgment result by detection result 
difference judging section 2F is negation, head movement magnitude calculation section 2G It has 
composition which computes the movement magnitude (on an eyeball image) (deltax, delta y) of the 
pupil center position by motion of a head in the form [ like ] which carries out inverse transformation 
processing of the detection result by cornea reflection method operation part 2E, and is sent into pupil 
core method operation part 2D. 

[0032] As shown in drawing 3 , when it is more specifically pupil center position Ps= (xc, yc) on the 
eyeball image PB in the case of pupil core method operation part 2D, as shown in drawing 2 , according 
to the following operation expression (1), look location Pt= (Xw, Yw) in Screen PA for alter operation is 
computed. The matrix which uses ABCD of operation expression (1) as an element corresponds to the 
correlation between the pupil center position Ps on the eyeball image PB called for as mentioned above 
based on four standard point Pa-Pd in Screen PA for alter operation, and the look location Pt on Screen 
PA for alter operation (correspondence relation). 
;0033] 
Equation 1] 
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[0034] When movement magnitude deltaP= (deltax, delta y) of the pupil center position by motion of a 
head is sent into pupil core method operation part 2D from head movement magnitude calculation 
section 2G, however, the pupil center position Ps Since only the movement magnitude deltaP of a pupil 
center position will have shifted from original pupil center position Pu= (xr, yr) according to the amount 
by which the head moved, head movement magnitude calculation section 2G The re-operation which 
computes the exact look location PT (Xv, Yv) where gap of movement magnitude deltaP was taken into 
consideration, and the error by motion of a head was eliminated according to the following operation 
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expression (2) will be performed. In addition, it is xr=xc+delta x and yr=yc+deltay and movement 
magnitude deltaP= (deltax, delta y) becomes forward or a negative-number value according to the sense 
of a coordinate. 

[0035] In addition, the look location for which it reasked by pupil core method operation part 2D is sent 
out to a personal computer 3. Therefore, in the case of the example, a detection result re-operation 
means to research for the detection result by pupil core method operation means by which the error 
resulting from a motion of the head which has an eyeball for detection based on the detection result of 
cornea reflection method operation part 2E was removed will consist of head movement magnitude 
calculation section 2G and pupil core method operation part 2D. 
;0036] 
Equation 2] 
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[0037] Moreover, look detection equipment 2 is equipped with blink detection section 2H which detect 
Operator M having winked and having carried out from the detection result by pupil core method 
operation part 2D, and blink detection section 2H send out the signal which winks to a personal 
computer 3 immediately and tells generating, if the blink of Operator's M eye is detected. 
[0038] On the other hand, according to the look location sent out from pupil core method operation part 
2D, a personal computer 3 indicates by superposition the cursor-like marker mk who shows a look 
location to Screen PA for alter operation of the head UMANTO display 4, as shown in drawing 4 . 
Marker mk has composition which moves in the Screen PA top for alter operation according to look 
change of Operator M. On the other hand, if the signal which a personal computer 3 winks from blink 
detection section 2H, and tells generating is received, as shown in drawing 5 , it will be judged that alter 
operation was performed to the actuation key (or icon) nk in which Marker mk is located. 
[0039] thus, since in the case of the system of an example Operator M winks with a look, it comes out 
and alter operation of a personal computer 3 can be performed, it becomes possible to communicate with 
a personal computer 3, while the serious trouble back tone who cannot do actuation of a keyboard at all 
had also gone to sleep (dialogue). 

[0040] Then, the activation process of communication by the system of the example which has the 
configuration described above is explained, referring to a drawing. Drawing 6 is a flow chart which 
shows the advance situation of communication by the system of an example. 
[0041] In addition, below, after Operator M is equipped with a head mount display 4, it explains as a 
thing in the phase which the matrix which uses ABCD as a multiplier equivalent to the correlation of the 
look location and pupil center position on Screen PA for alter operation as an element was already **** 
(ed), and all preparations finished. 

[0042] [Step SI] Operator's M eyeball is photoed by the optical image pick-up device 1, and an eyeball 
image is sent into look detection equipment 2. 

[0043] [Step S2] After the eyeball image captured by 3 value-ized circuit 2A is formed into 3 values, 
while a pupil center position is called for by pupil extract section 2B from 3 value-ized image, the 
location of a reflected image is called for by reflected image extract section 2B. 
[0044] [Step S3] While a look location is called for from a pupil center position by pupil core method 
operation part 2D, a look location is detected by corneal-reflex operation part 2E from a pupil center 
position and the location of a reflected image. 

[0045] [Step S4] The difference of both operation part 2D and the detection result of the look location of 
2E carries out fixed time amount continuation by detection result difference judging section 2F, and it is 
judged whether it is predetermined within the limits. If a judgment result is affirmation, it will fly to step 
S7, and if a judgment result is negation, it will progress to the following step S5. 
[0046] [Step S5] Movement magnitude deltaP based on [ by motion of Operator's M head ] pupils is 
****(ed) by head movement magnitude calculation section 2G. 
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[0047] [Step S6] Pupil core method operation part 2D takes into consideration movement magnitude 
deltaP based on [ by motion of Operator's M head ] pupils, and re**** the look location PT. 
[0048] [Step S7] The look location Pt or the look location PT ****(ed) by pupil core method operation 
part 2D is sent into a personal computer 3. 

[0049] [Step S8] When a blink of Operator M is detected by blink detecting-element 2H, it progresses to 
the following step S9, and when a blink of Operator M is not detected, it flies to step S10. 
[0050] [Step S9] The signal which winks from blink detecting-element 2H, and tells generating is sent 
into a personal computer 3, and alter operation corresponding to a look location is performed. 
[0051] [Step S10] When continuing detection of a look location, return and the following steps are 
repeated to step SI. Communication is ended when not continuing detection of a look location. 
[0052] It is not restored to predetermined within the limits which the difference of each two detection 
results computed by each algorithm of a pupil core method and a pupil-cornea reflection method to the 
captured eyeball image in the example set up beforehand as stated above. When the error resulting from 
a motion of Operator's M head exists, it has the composition that the re-operation which reasks for the 
look location except the error which originates in a motion of Operator's M head in quest of movement 
magnitude deltaP of the pupil center position by motion of a head is performed. Consequently, exact 
detection by the algorithm of a pupil core method is always performed, and the look location where 
Operator's M eyeball is exact can always detect, without being influenced by the installation location of 
the angle of rotation of an eyeball, or the illumination light for an image pick-up. 
[0053] This invention is not restricted to the gestalt of the above-mentioned implementation, and can 
carry out deformation implementation as follows. 

[0054] (1) If required location detection is possible, you may make it adopt in the example, the 
configuration which binary-ization-processes an eyeball image, although it was the configuration of 3- 
value-ization-processing.an eyeball image. 

[0055] (2) In the example, although it was the configuration of deciding a look location because there 
was a blink, the thing of a configuration of deciding a look location by switch actuation or look duration 
is mentioned as a modification. 

[0056] (3) In the case of the example, it was the configuration that the detection result of a look location 
was used for the alter operation of a computer, but in this invention, the detection result of a look 
location may be a configuration used for things other than the alter operation of a computer, and may be 
only a configuration which only searches for the detection result of a look location. 



[Translation done.] 



http://www4.ipdl.ncipi.go.jp/cgi-bin/tran_web_cgi_ejje 



3/14/07 



* NOTICES * 



JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2 **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the block diagram showing the whole computer communication system configuration 
using an example of the look detection equipment of this invention. 

[Drawing 2] It is drawing showing an example of the screen for alter operation of the computer of the 
system of an example. 

[Drawing 3] It is drawing showing change of the pupil center position produced in connection with a 
motion of an operator head in the eyeball image of an example. 

[Drawing 4] It is drawing showing other examples of the screen for alter operation of the computer of 
the system of an example. 

[Drawing 5] It is drawing showing other examples of the screen for alter operation of the computer of 
the system of an example. 

[Drawing 6] It is the flow chart which shows the advance situation of communication by the system of 
an example. 

[Description of Notations] 

1 - Optical Image Pick-up Device 

2 — Look Detection Equipment 

2D — Pupil core method operation part 
2E - Cornea reflection method operation part 
2F - Detection result difference judging section 
2G - Head movement magnitude calculation section 

3 - Personal Computer 

4 - Head Mount Display 

PA — Screen for alter operation 
PT, Pt — Look location 
M — Operator 
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DRAWINGS 



[Drawing 1] 
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[Drawing 3] 




[Drawing 4] 
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[Drawing 6] 
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